Over the last decade, malicious software has become a pervasive problem for Internet users as many networked resources include vulnerabilities that are subject to attack. For instance, over the past few years, more and more vulnerabilities are being discovered in software that is loaded onto network devices, such as vulnerabilities within operating systems for example. While some vulnerabilities continue to be addressed through software patches, prior to the release of such software patches, network devices will continue to be targeted for attack by exploits, namely malicious computer code that attempts to take advantage of a vulnerability in computer software by acquiring sensitive information or adversely influencing or attacking normal operations of the network device or the entire enterprise network.
In particular, an exploitation technique known as return-oriented programming (ROP) has become fairly widespread recently. ROP is an exploitation technique that allows a writer of malware to chain together sequences of instructions through “return” instructions thereby accomplishing one or more tasks via the execution of the chain of sequences of instructions. The ROP technique was developed as a way to circumvent exploit prevention techniques, such as data execution prevention (DEP) techniques, which have been recently implemented in many operating systems to thwart unauthorized activities including malicious attacks.
Using the ROP technique, malware writers attempt to gain control of the stack and subsequently execute sequences of instructions appearing in executable code, such as a dynamically-loaded library (DLL). The sequences of instructions are chained together through the use of “return” instructions following the sequence of instructions. For example, the “return” instruction following a first sequence of instructions (sequence_1) will point to a starting address of a second sequence of instructions (sequence_2).
The ROP technique often violates a well-known programming invariant which states that an instruction immediately preceding the location branched to by a “return” instruction must be a “call” instruction. Therefore, a malware detection system may detect the use of the ROP technique by determining whether the execution of an application violates this invariant.
Currently, malware detection systems running within a virtual machine may attempt to detect the use of the ROP technique by analyzing the Last Branch Records (LBRs) of a CPU while an application under analysis is executing. However, the application may thwart or avoid detection by reading a register of the model-specific registers (MSRs) and thereby detecting that the LBR functionality is turned on. The application may then alter its intended malicious operations to avoid detection. In addition, the application may detect the LBR functionality is turned on and subsequently turn off the LBR functionality.